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1. Introduction 

Early research on the development of machine vision algorithms to serve as pilot aids in 
aircraft flight operations is discussed. The research is useful for synthesizing new cockpit 
instrumentation that can enhance flight safety and efficiency. With the present work as the basis, 
future research will produce low-cost instrument by integrating a conventional TV camera together 
with off-the-shelf digitizing hardware for flight test verification. Initial focus of the research will be 
on developing pilot aids for clear-night operations. Latter part of the research will examine 
synthetic vision issues for poor visibility flight operations. Both research efforts will contribute 
towards the high-speed civil transport aircraft program. 

It is anticipated that the research reported here will also produce pilot aids for conducting 
helicopter flight operations during emergency search and rescue. The primary emphasis of the 
present research effort is on near-term, flight demonstrable technologies. The following sections 
will discuss each of the research items in further detail. 

2. Pilot Aids for Night Landing and Takeoff 

Various difficulties experienced by the pilots while executing night landing and takeoff 
operations are well documented [1], Altimeter, Pitot sensor. Gyrocompass, attitude indicator, and 
a radio are sufficient to aid the pilot in aircraft control and navigation during cruise flight at night. 
However, these instruments are not useful for night time landing, even under good visibility 
conditions. Pilots have to depend on runway lights to provide the appropriate landing-takeoff 
cues. 

Reference 1 documents the several dangerous optical illusions that can arise during 
nighttime landing operations. Most of these arise due to the absence of scene shading, which often 
makes it difficult to perceive the depth of the visual field. Consequently, most pilots are 
uncomfortable landing at an unfamiliar airfield at night. Night operational problems include 
difficulty in judging the range, altitude, attitude and runway limits when observed from the air. 

Additional difficulties arise due to the complex approach procedures required at many 
airports. Airports near population centers often have flight hazards such as tall buildings and 
transmission towers. Restrictions arising from noise abatement regulations can further contribute 
towards operational difficulties. Typically, aircraft are required to employ a steeper approach path 
before capturing the glide slope towards the runway. At night, this means that the pilot must be 
able to accurately judge the distance to the airport throughout the landing and takeoff phases. 
Operating under such conditions is a high-stress activity for the pilot because any deviation from 
the prescribed procedure can have disastrous consequences. 

Advanced guidance systems such as the microwave landing system (MLS) can obviate 
these difficulties. However, such systems are expensive and are not likely to find widespread 
application in smaller airports. Guidance methods that do not need expensive antennas and 
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transmitters are likely tind higher levels of acceptance from the aviation community. Thus, runway 
lighting systems will continue to be the chief landing aids for night flying. Federal Aviation 
Administration has recognized this fact, and is investing in the deployment of various visual 
landing aids. Recendy deployed PAPI system [2] is an example of this initiative. 

The objective of the present research is to further increase the utility of the existing runway 
lighdng systems through the use of an on-board machine vision system. Machine vision systems 
have shown excellent promise in the development of rotorcraft nap-of-the-earth flight navigation 
systems [3, 4], The main characteristic of such systems is their ability to generate position of 
various objects within the held-of-view with respect to the vehicle using an image sequence. 
Conversely, they can be used to determine the position of a flight vehicle with respect to specific 
objects within the held-of-view. The problem considered in this proposal is considerably simpler 
than those in References 3 and 4, because the runway lighting system has a well defined structure. 

One of the objectives of the present research is to develop an automatic machine-vision 
based system that will use the image sequence from an aircraft mounted camera to compute down- 
range, cross-range, altitude, and glide slope. These data can be displayed to the pilot in a suitable 
form. Pilot display may include range to go, desired Vs actual glide slope, cross range error with 
respect to the runway centerline etc. In other words, the present research attempts to develop a 
vision-based equivalent of a microwave landing system (MLS). 

Algorithms will be synthesized to determine the aircraft position, attitude, and velocity 
vectors with respect to an airport equipped with a standard runway lighting system. An off-the- 
shelf medium to low resolution TV camera together with a low-cost digitizer will be used to collect 
images of the airport. These images will form the input to the machine vision algorithm 
implemented on a personal computer. The computer outputs will be displayed to the pilot in the 
form of a guidance display. 

The vision system will also find extensive use in helicopter flight operations. Specifically, 
the system can be used to aid the pilot to precisely align and determine the distance to the landing 
area. Note that helicopter pilots have difficulty in seeing the landing area when they are close to the 
ground. This can be hazardous when the helicopters are trying to land in relatively unfamiliar and 
unprepared surface. A methodology to aid helicopter pilots in such situations will be developed as 
a part of future research. 

We expect a machine vision-based instrument to provide accurate navigational information 
to the pilot. Initial studies indicate that such an instrument can be very accurate. Two distinct 
approaches have been explored to date. In the first approach, aircraft altimeter data is used to 
determine the down-range and cross range locations of various runway lights. These computations 
require the knowledge of camera/aircraft attitude with respect to the runway. When implemented, 
this algorithm can be yield the aircraft glide slope, and the aircraft heading angle orientation with 
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respect to the runway. The second approach attempts to compute the aircraft position with respect 
to the runway without using the altimeter data. The computational method in this case turns out to 
be iterative. The well known Newton-Ralphson method proved to be very effective for this 
problem. However, the procedure still requires the aircraft attitude data. We believe that the 
computational procedure can be refined to a point where it would possible to obtain all the runway 
related aircraft state variables for pilot display and guidance. 

Results from this research are being written-up as a paper for the 1994 AIAA Guidance, 
Navigation, and Control Conference. 

2.1. Future Research on Pilot Aids for Night Landing and Takeoff 

With the foregoing, the following research directions are proposed for the next three year 

period: 

1. Develop the vision-based aircraft position determination method using all the available 
information about the runway lighting system. 

2. Investigate methods for vision-based aircraft formation keeping, rendezvous, and avoidance. 

3. Develop a pilot assisted guidance scheme to enable the optimum use of the navigation data. 

4. Demonstrate the performance of the algorithm using a scene simulation on a 3-D Graphics 
computer. 

5. Configure a flight test system using off-the-shelf video and computing hardware. 

Periodic presentations, oral and written reports will be provided at the end of each research 

period. 

3. Synthetic Vision as an Aid to Low Visibility Landing 

Development of synthetic vision aids is considered to be the next step in improving airline 
operational efficiency [5-7]. Although microwave landing systems are the state-of-the-art for 
extremely low visibility operations, these systems require significant investment both on the 
ground and on each aircraft. Secondly, although these guidance systems are effective in helping 
aircraft land, they are not useful in ground operations such as taxi, and terminal approach. This fact 
forces the airport to operate at significantly reduced capacity. This forms one of the motivating 
factors for the development of a synthetic vision system. Another motivating factor is the recent 
research initiative on high-speed civil transport program. Due to their relatively inefficient low 
speed performance, fixed-configuration supersonic aircraft have to maintain a high angle of attack 
during approach and landing. Since these aircraft configurations have long nose sections, this 
implies that the pilots will be unable to see the ground during landing. Aircraft such as the Anglo- 
French Concord employ an articulated nose section to improve pilot visibility. However, recent 
studies show that avoiding drooped-nose configuration could save between 1500 and 2000 pounds 
of structure weight, and reduce gross takeoff weight as much as 10, 000 pounds. Synthetic vision 
can be used to avoid the use of the articulated nose section, thereby increasing the system 
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efficiency. Finally, if a synthetic vision system permits reduced-visibility weather conditions, the 
aircraft could carry less reserve fuel for diversion purposes. Less fuel leads to smaller size aircraft, 
which can further reduce the airframe weight when compared with cargo/passenger load. 

Synthetic vision concept currently under development at various aeronautical research 
centers focus on generating out-of-the-window scene by combining information from various 
sensors. For instance, images obtained from low-light level TV cameras can be combined with 
those obtained using infrared cameras, and passive millimeter wave radar to generate an integrated 
display system. The image displayed to the pilot may include various levels of detail. At a very 
basic level, a wire frame model of the airport and the surrounding area may be displayed to the 
pilot. More advanced systems can generate and display fully shaded color images of the scene. 

The fundamental principle that aids in generating synthetic vision displays is the fact that 
the scene geometry forms unifying basis for all the images. Geometric data of most major airports 
are currently available as computer data bases used in training simulators. The challenge is in 
synthesizing algorithms that can combine this data with those from other sensors to form a display 
system that can augment pilot effectiveness. 

Initial research into such algorithms is given in Appendix - A. That research will 
demonstrate the fusion of geometric data with a gray scale image. The method is based on the 
classical predictor-corrector paradigm. The geometry data together with an initial image are used to 
form a predicted image. Whenever the actual image is obtained, the error between the predicted and 
actual images are used to update the initial image. The updating process can include replacing 
areas of the actual image by stored texture maps corrected for illumination conditions. The 
assumption in such an approach is that the geometric data is accurate, while the actual images 
contain various errors due to imaging sensor characteristics. 

The material presented in Appendix-A illustrates the development of an image predictor that 
combines an image with the corresponding geometric data to form a new image. Measured image 
can be combined with the predicted image to form an improved image. The subsystem for 
performing image correction is currently under development. As a» initial application of the 
predictor-corrector paradigm. Appendix - A discusses an image-based ranging scheme for ANOE 
flight. 

3.1. Future Research on Synthetic Vision 

In view of the early research results using the predictor-corrector paradigm, the following 
research directions are proposed for the next research phase. 

1. Investigate alternate methods for image prediction. 

2. Formulate a corrector scheme for updating the image. 

3. Generate corrupted images by superimposing random patches of textures on laboratory image 
sequences. Generate simulated images for Passive millimeter wave radar and Infrared sensors. 
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4. Using these images, demonstrate the performance of the predictor-corrector method for 
synthetic image generation. 

Progress on these and other related research items will be reported using periodic presentations and 
reports. 
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Abstract 

Synthetic Vision systems are considered to be a crucial element in the operation of next 
generation transport aircraft This appendix reports on the progress in combining known geometric 
data with an image sequence of the scene. Proposed procedure employs the predictor-corrector 
paradigm. As an application, the use of the proposed method for developing a field-based ranging 
algorithm is also discussed. Convergence properties are examined using a simplified analysis. 
Introduction 

Two approaches have been advanced for guiding next generation aircraft under Category 
III flight conditions. First of these is the Microwave Landing System (MLS). Synthetic vision 
forms the second approach. In the first approach a microwave beacon is used to guide the aircraft 
to the touch-down point, and in initiating the flare maneuver. It does not aid the aircraft taxi, or 
maneuver in the terminal area. The system operates in a heads-down mode, with the pilot using the 
flight director display to operate the aircraft. These systems have been around for a number of 
years, and are considered reliable. 

Synthetic vision approaches, on the other hand, focus on a heads-up mode by displaying a 
synthetically generated imagery to the pilot. The imagery is generated by combining computer data 
base with the data obtained from several sensors, most of which operate in a passive fashion. 
Typical sensors include passive millimeter wave radar, low-light TV, and infrared imaging 
systems. 

While most of the MLS equipment is on the ground, with the aircraft carrying only a 
receiver, synthetic vision approach is an aircraft based solution. All the equipment for the synthetic 
vision will be carried on-board the aircraft, including the computer data bases. 

The major challenge for synthetic vision is in developing computer algorithms that can 
accept data from disparate sources and generate an intuitively obvious visual display for pilot's 
use. Computer data bases generally contain three-dimensional data about various features on the 
airfield under consideration. Various sensors produce segments of the field-of-view, which have to 
be combined together to form an image that the pilot can use to land the aircraft. Visual data from 
these sensors generally do not have sufficient resolution or detail to be directly useful to the pilot 

This appendix discuses early research experiments on synthetic vision. The predictor- 
corrector paradigm forms the basis for the system. An image predictor and one of its applications 
will be discussed in the following sections. Corrector algorithms are currently under development. 
Image Predictor 

Image predictor uses an image and the associated geometric data to form a new image based 
on the camera movement. The image predictor takes an image specified as a two-dimensional array 
E(i, j), together with a depth array Z(i, j) and vehicle/camera motion parameters to come up with a 
predicted image Ep(i, j). Only certain parts of the images can be predicted because of the limitations 
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in the camera field of view. Regions where it is not feasible to predict, segments of the actual 
image will have to be used. 

Image prediction method will depend upon the type of processing that has been carried out 
on the original image. This fact motivates the analysis of the imaging process. Digital images are 
obtained by processing continuous images through a spatio-temporal sample-hold operation. Gray- 
scale image display systems on computer normally employ uniform irradiance in a pixel, 
effectively implementing a zeroth-order hold. If the image sampling is carried out at a frequency 
satisfying the sampling theorem, continuous image can be recovered from the discrete 
representation using various hold functions. However, the purpose to which the resulting image is 
put to determines the type of hold function that should be implemented. For instance, if the 
objective is to zoom-in, zeroth-order hold would yield an image of lower resolution [1], A first- 
order hold is a superior choice in this situation. On the other hand, if the objective is to zoom-out, 
zeroth-order hold will be adequate to preserve the resolution. Various steps involved implementing 
an image manipulation system is shown in Figure 1. 



Fig. 1. Image Manipulation System 

A digital camera produces a sampled image for use with various other subsystems. The image can 
be displayed using various hold functions. Hold function converts the digital image into a 
continuous form. Continuous images can be rotated and translated depending upon the camera 
motion. The resulting continuous image can then be re-sampled. If the original spatio-temporal 
sampling rate satisfies the sampling theorem [1], the manipulated image will not contain any 
distortion. However, the type of hold function implemented in the system can contribute to loss of 
resolution. 

An issue that needs to be resolved before implementing an image predictor is that of the 
scene depth distribution. Clearly, the scene depth can vary dramatically from pixel to pixel. 
Depending upon the distance of the object from the camera, the scene depth could vary even within 
a pixel. However, since a pixel is the smallest unit of the image, it will be assumed that the scene 
depth is uniform over the entire pixel area. Although logically correct, this assumption can lead to 


8 




scene depth displaying jagged edges for objects that are close to the camera. If this problem is 
serious, the jagged edges can be eliminated only by using a camera with improved resolution. 

With this background, the prediction problem may be defined as: Given an image ( Pixel array 
consisting of the irradiance at each pixel), camera focal length, focal plane array dimensions or the 
field-of-view, camera rotation and translation with respect to a pre-specified datum, and depth at 
each pixel, predict the next image. 

Denote a specific pixel in the image plane by P(i, j). Let the row dimension of the image 
plane be n and the column dimension be m. Let the field-of view (in radians) along the horizontal 
dimension be d and that along the vertical dimension be vv. Thus, the dimensions of a pixel are 
(df)ln x ( wf)lm . Assume that the image plane coordinate system is located at the center of the 
picture. Thus, the picture dimensions with respect to the origin are: - (df)/2, (df)/ 2, - (wf)/ 2, 
(wf)/2. 

At a particular pixel P(i. j), with i, j starting at the top left hand comer of the picture, 

perspective projection geometry can be used to relate the position coordinates of the visual field 

with the pixel location. Horizontal and vertical location of the four comers of the pixel with respect 
to the origin of the image coordinate system can be obtained as: 

Top left comer : df[{ i - 1 )/n - 1/2], wf [(j -1 )tm - 1/2] 

Top right comer: df [ tin - 1/2], wf [(j -1 )/m - 1/2] 

Bottom left comer : df[{ i - l)/n - 1/2], wf[j/m- 1/2] 

Bottom right comer : df[ iln - 1/2], w/[j lm - 1/2] 

Pixel center : df[( i - 1/2 )/n - 1/2], w/[(j -1/2 )/m - 1/2] 

The discrete sample will be assumed to be located at the pixel center. Let xy, yy, zy be the 
inertial coordinates of an object at pixel location (i, j). It can be shown [2] that the inertial position 
coordinates can be related to the camera position coordinates as: 

x c = T i T 2 (x-Xv)-Ti I 

In response to the vehicle motion by Ax v the position of objects in the camera coordinate system 
will change to: 

Ax c = T i AT2 (x - x v ) - Ti T2 x v 
x c + Ax c = Ti (T 2 + AT 2 ) (x - x v ) - Ti I - Ti T 2 x v 
If the components of the vectors (x c + Ax c ) and Ax c are known, the pixel displacement at any 
pixel location xp, yp can be computed as: 

Axp = f Ay c /(x c +Ax c ) - x p Ax c /(x c +Ax c ) 

Ayp = t Az c /(x c +Ax c ) - yp Axc/(xc+Axc) 
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The pixel displacement can be used to predict the image. Due to the fact that depths at each pixel are 
difterent. the new image will not be on a uniform grid. Moreover, there may be areas where the 
image "tears", i. e. none of the image intensities may map to a certain areas of the imaee. 
Additionally, we may have areas where the predicted image "tolds”, i.e. areas where several pixels 
would get mapped into. In the case of folding, it is clear that the pixels corresponding to objects 
behind the pixel under consideration can be deleted because all the objects are assumed to be 
opaque. Treatment ot tears, however, is not direct. Although irradiance interpolation using nearby 
pixels appears to be an obvious approach, it is not desirable. This is because of the fact that the 
areas where the image "tears” correspond to occluding objects moving away from the line of sight 
due to camera motion. Whenever the image tears, it is better to avoid prediction and use the actual 
image. 

To summarize, the image prediction process proceeds as follows: 

1. Based on an initial knowledge of the depth map x, and vehicle-camera motion x v , compute 
(xc+Axc) and Ax<;. Compute the pixel displacements Axp, Ayp at each pixel. 

2. Transform the current sampled image using the computed pixel displacements. This will yield a 
sampled image on a non-uniform grid. Discard pixels based on which pixel is in front of the other 
pixels mapped to the same location. 

3. Assuming that the sampling theorem is satisfied, convert a sampled image to continuous form 
using a "hold” function. First-order hold will yield acceptable fidelity in most digital image 
sequences. 

4. Sample the transformed continuous image on the original uniform grid to generate the predicted 
sampled image. 

5. Assuming that the scene depth is constant over an individual pixel area, transform the range map 
also. 

6. Fill-in areas of "tears” using the actual image. 

Note that the prediction process does not have to be perfect for the method to work satisfactorily. 
The correction algorithm under development will ensure that the predicted image remains close to 
the actual throughout the process. 

Linear Prediction 

The image prediction process described in the foregoing can demand excessive computing 
time. If the temporal sampling rate of the imaging process is sufficiently high, the differences in 
images in a sequence will be small. Under this condition, it is feasible to employ a linear image 
prediction technique. Taylor series expansion of an image about each pixel forms the underlying 
principle tor such a process. Taylor series expansion can be combined with perspective projection 
relations to form image-based ranging equation given in Reference 2. If the camera undergoes a 
pure translation, the image-based ranging equation will be as follows: 
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E2 is the predicted image, Ej is the initial image and the scene depth is z. The camera is assumed 
to have undergone a translational motion ot Ax(), Ayo, Azo. The initial location of the camera is 
assumed to be x(), yO * W- The variables xp, yp are the pixel location in the image plane, and f is 
the camera focal length. 

This equation can be used perform rapid image prediction. Due to the nature of the linear 
prediction process, difficulties such as tearing and folding will not arise. This is not necessarily 
good, since occlusion is a real phenomenon that needs to be included in the image prediction 
process. Thus, in order to maintain fidelity, it is important to periodically replace predicted images 
with actual images. 


The image predictor can be combined with a correction algorithm to generate a synthetic 
vision system. The corrector has the responsibility for ensuring the consistency between the scene 
geometry and the corresponding image. Various aspects of the corrector will be pursued in the 
ensuing research period. The utility ot the image predictor discussed in the foregoing can be 
illustrated using an example. In the following subsection, we will illustrate the use of the image 
predictor for constructing an image-based ranging scheme. 

A Predictor-Corrector Algorithm for Image-Based Ranging 


The development will employ the vision-based ranging equation developed in Reference 2. 
The ranging equation from Reference 2 is: 


E 2 = Ei + 


^ [xpAz 0 - f Ax 0 ] + —[ypAz 0 - f Ay 0 ] 
ax n dy P 


1 


z - zq - Az 0 


In this equation, E\ is the first image, E2 the second image, x p , y p are the pixel locations, xo, yo, 
zo are the vehicle position components, Ax(), Ayo, Azo changes in the vehicle/camera position 
components, and z is the scene depth, the only unknown in this equation. 

The ranging equation can be further simplified by invoking the following assumptions: 

a) The image is assumed to consist of constant gradient patches. 

b) The image baseline is assumed to be fixed between images. 

Under these assumptions, the Image-based ranging equation can be written in a more concise form 
as: 


E 2 - E[ = K 8 

Here, K is the quantity within the large square brackets, and 8 is the inverse depth of the scene. 
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Initial Image 


Fig. 4. Predictor- Corrector Scheme for Image-Based Ranging 

This expression can be used to formulate the predictor-corrector technique. Various steps in this 
procedure are given in the following: 

/N 

1. Image Prediction: Starting with the image El, estimated scene depth 8, the predicted image E 2 is 

E 2 = Ei + K 8 

2. Let the measured image E2- If we knew the exact scene depth 5, the measured image would 
have been: E 2 = Ei + K 8. The error in inverse of scene depth can be obtained by subtracting the 
predicted image from the actual as: 

E 2 - E 2 = K (S - 5) = K e 

3. The scene depth error can then be computed as: 

c _ E 2 - E 2 

K 

4. Correct the depth estimate using computed depth error: 

/\ 

5i+i = 8; + a e 

5. Correct the original image using the error between predicted image and the measured image.. 


12 





Ei =E, +3 Ke 


6. Return to step 1. 

Fig. 4. gives a block diagram of the simplified predictor-corrector scheme. Note that if K is 
constant, the block diagram represents a linear, time-invariant system. 

Using elementary block diagram manipulations, it is possible to show that the system will 
be stable. Specifically, it is possible to demonstrate the following: 

1. Starting with an arbitrary initial guess of the inverse scene depth, it can be shown that the 
inverse depth estimates uniformly go to zero if the pixel intensity remains constant, (motion implies 
a non zero value of K). Note that this situation corresponds to an object located at infinity. 

2. If the camera is moving towards the scene at a constant rate, since the scene patch has been 
assumed to have constant gradient, the measured irradiance will uniformly increase at any pixel 
other than the one at the center of the image. In this case, it can be shown that the predicted image 
irradiance and the actual image irradiance will closely track each other. 

Conclusions 

Initial research on combining geometry ot scene with the corresponding image was given 
Various tradeoffs involved in the synthesis of an image predictor was outlined. Linearized 
approaches to the problem were examined. The utility of the predictor in an image-based ranging 
scheme was illustrated. 

Next Research phase will focus on the synthesis of a corrector algorithm. A synthetic 
vision algorithm using the predictor-corrector algorithm will also be demonstrated. 
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